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DATA PROCESSING APPARATUS AND METHOD 



AND 



COMPUTER READABLE STORAGE MEDIUM 



BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to a data processing 
apparatus and method for encoding, multiplexing and 
outputting digitalized video and audio data, and a 
computer readable storage medium storing data 
processing programs realizing the data processing 
method . 

Related Background Art 

As international standards for a coding system of 
video, audio and other data, JPEG (Joint Photographic 
Coding Experts Group), H. 261 and MPEG (Moving Picture 
Coding Experts Group) which is an improved version of 
JPEG and H. 261 are known. 

In the present days called a multimedia period 
collectively dealing with video, audio and other data, 
MPEG-1 as improved MPEG and further MPEG- 2 as improved 
MPEG-1 are adopted. 

Of these specifications, MPEG-2 is specifications 
which are recommended for transmitting video, audio and 
other data at a high quality in a limited transmission 
band. Equipments compatible with the specifications of 
MPEG-2 are in the main stream. For example, with MPEG- 
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2, not only a desired number of respective encoded 
streams of video, audio and other data can be 
multiplexed into one stream (PS: Program Stream) as a 
set of programs, but also a plurality of programs can 
5 be constituted as one stream ( TS : Transport Stream). 

In a digital transmission apparatus compatible 
with the specifications of MPEG-2, an encoder unit of 
the apparatus encodes audio and video data and 
packetizes it into a PES (Packetized Elementary Stream 

10 which is again packetized into a TS packet and 
transmitted . 

Packetizing will be specifically described by 
paying attention to audio data. 

An audio data compression circuit of a digital 

15 transmission apparatus first compresses audio data in 
accordance with a predetermined scheme (e.g., ISO/IEC 
11172-3), to form AAU's (Audio Access Units) each 
having a compression/expansion base unit and output an 
AES (Audio Elementary Stream) constituted of AAU's. 

20 More specifically, as shown in Fig. 1A, AES 

constituted of AAUO, AAU1, ... is output. 

Next, as shown in Fig. IB, AES is packetized in 
the unit of AAU , and a PTS (Presentation Time Stamp) is 
loaded in a header field (PES header) of each packet 

25 (PES packet). This packet as an audio PES is passed to 

the next process. According to ISO/IEC 13818-1, it 
stipulates that although it is not necessarily required 



to load PTS in each PES header, PTS is required to be 
loaded and transmitted at least once per 700 ms. 

At the next process, as shown in Fig. 1C, each PES 
packet is further packetized into a transmission packet 
(TS packet) to make it have a fixed length size 
suitable for a transmission path. In order to make 
each packet have the same packet length, invalid data 
(staff data) is added if necessary as shown in Fig. 1C. 

In a header field ( TSP header) of a TS packet, a 
PID (Packet Identification) for identifying data (such 
as audio data and video data) stored in the packet, a 
PCR (Program Clock Reference) and the like are loaded. 
According to ISO/IEC 13818-1, it stipulates that 
although it is not necessarily required to load PCR in 
each TSP header, PCR is required to be loaded and 
transmitted at least once per 100 ms. 

As above, audio data is transmitted as a TS 
packet . 

AAU is the minimum unit which can be recovered 
independently, and is always constituted of data having 
a constant number of samples. Namely, AAU is 
constituted of data sampled during a predetermined 
period and having a fixed length preset for each 
compression factor . 

PES including AAU ! s is packetized into each PES 
packet in the unit of AAU, and each PES packet is 
further packetized (divided) into a fixed length TS 



packet. In this case, it is not guaranteed that the 
data length of AAU , i.e., the data length of the PES 
packet, is an integer multiple of the data length of 
the TS packet. When each PES packet of PES is 
packetized and if the data of the PES packet is 
insufficient for the TS packet, the staff data is 
inserted into the last TS packet to make all TS packets 
have the same packet length. 

As above, the data length of AAU (data length of 
the PES packet) has no definite relation to the data 
length of the TS packet and hence to the data length of 
the last TS packet. Therefore, the length of the staff 
data to be inserted so as to make the last TS packet 
have the fixed length is uncertain, which results in a 
reduction of the capacity of a transmission path. 

SUMMARY OF THE INVENTION 

Under the above-described background, it is an 
object of the present invention to provide a data 
processing apparatus and method capable of improving 
more than conventional an efficiency of a transmission 
path along which encoded data is transmitted, and a 
storage medium storing program software which executes 
the data processing method. 

According to one preferred embodiment, a data 
processing method is provided which comprises the steps 
of: inputting data; compression-encoding the data; 



packetizing the encoded data into a first data train on 
a basis of a first data length; packetizing the 
generated first data train into a second data train on 
a basis of a second data length; and controlling the 
first data length at said first packetizing step in 
accordance with the second data length. 

According to another preferred embodiment, a 
computer readable storage medium storing an image 
processing program is provided, the program comprising: 
an input step of inputting data; an encoding step of 
compression-encoding the data; a first packetizing step 
of packetizing the data encoded by said encoding step 
into a first data train on a basis of a first data 
length; a second packetizing step of packetizing the 
first data train generated by said first packetizing 
step into a second data train on a basis of a second 
data length; and a control step of controlling the 
first data length at said first packetizing step in 
accordance with the second data length. 

Other objects, features and advantages of the 
invention will become apparent from the following 
detailed description taken in conjunction with the 
accompanying drawings . 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figs. 1A, IB and 1C are diagrams illustrating 
packetizing of audio data. 
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Fig. 2 is comprised of Figs. 2A and 2B 
illustrating block diagrams showing the structure of a 
digital data transmission system according to an 
embodiment of the invention. 
5 Figs. 3A, 3B and 3C are diagrams illustrating 

packetizing of audio data according to the invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Embodiments of the invention will be described 

10 with the accompanying drawings. 

Figs. 2A and 2B are block diagrams showing the 
structure of a data processing system according to an 
embodiment of the invention. 

This data processing system is a system compatible 

15 with the specifications of MPEG-2. As shown in Figs. 

2A and 2B, the data processing system includes a camera 
unit 10, an encoder unit 100, a decoder unit 200 and a 
monitor unit 400. Video and audio data taken with the 
camera unit 10 is supplied to the encoder unit 100. 

20 The encoder unit 100 compresses and encodes the video 
and audio data and then packetizes it. The packetized 
data is transmitted via a transmission path 300 to the 
decoder unit 200. The decoder unit 200 decodes the 
encoded video and audio data and the monitor 400 

25 outputs decoded video and audio data. 

The camera unit 10 supplies video data of an image 
taken with an unrepresented camera and audio data 



# • 

- 7 - 



picked up with an unrepresented microphone to the 
encoder unit 100. 

The encoder unit 100 has: an input terminal 101 to 
which audio data supplied from the camera unit 10 is 
5 applied; an input terminal 102 to which video data 

supplied from the camera unit 10 is applied; an audio 
data compression circuit 104 to be supplied with an 
output from the input terminal 101; a video data 
compression circuit 106 to be supplied with an output 
q 10 from the input terminal 102; a compressed audio data 
ff* packeting circuit 107 to be supplied with an output 

^ from the audio data compression circuit 104; and a 

^ compressed video data packeting circuit 109 to be 

;f. t 

t "S3- 

HJ supplied with an output from the video data compression 

Q 15 circuit 106. 

5 Ft 

M: The encoder unit 100 has also: a multiplex 

yr§ transmission packeting circuit 110 to be supplied with 

~~ the outputs from the audio and video data packeting 

circuits 107 and 109; a modulation circuit 111 to be 
20 supplied with an output of the multiplex transmission 
packeting circuit 110; a compression control circuit 
105 for controlling the compression to be executed by 
the audio data compression circuit 104; a reference 
time generation circuit 103 for generating and 
25 supplying timing signals to the audio data compression 
circuit 104, video data compression circuit 106 and 
multiplex transmission packeting circuit 110. An 
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output of the modulation circuit 111 is supplied via a 
transmission path 300 to the decoder unit 200. 

The decoder unit 200 has: a demodulation circuit 
201 for receiving data from the transmission path; a 
5 multiplexed data separation circuit 202 for receiving 
data from the demodulation circuit 201; a compressed 
audio data depacketing circuit 203 and a compressed 
video data depacketing circuit 204 respectively 
receiving data from the multiplexed data separation 
o 10 circuit 202; a reference time recovery circuit 205; an 

audio data decoding circuit 206 to be supplied with 
outputs from the compressed audio data depacketing 
circuit 203 and reference time recovery circuit 205; 
and a video data decoding circuit 207 to be supplied 
15 with outputs from the compressed video data depacketing 
circuit 204 and reference time recovery circuit 205. 
An output of the audio data decoding circuit 206 is 
output via an output terminal 208, and an output of the 
video data decoding circuit 207 is output via an output 
20 terminal 209. Outputs from the output terminals 208 

and 209 are supplied to the monitor unit 400 which 
outputs images and sounds . 

The most characteristic structure of the digital 
transmission system described above is a packet size 
25 control circuit 108 provided in the encoder unit 100. 

The packet size control circuit 108 controls the 
audio data compression circuit 108 which is controlled 
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also by the compression control circuit 105, and also 
controls the compressed audio data packeting circuit 
107. 

A series of operations of the digital transmission 
5 system will be described first. 

Audio and video data output from the camera unit 
10 is applied to the input terminals 101 and 102. 

In the encoder unit 100, analog audio data is 
applied to the input terminal 101, whereas analog video 
10 data is applied to the input terminal 102. 

The reference time generator circuit 103 supplies 
the multiplex transmission packeting circuit 110 with a 
time of an audio signal input to the input terminal 101 
and a time of a video signal input to the input 
15 terminal 102, as program clock reference (PCR) values. 

The reference time generation circuit 103 also supplies 
the audio and video signal compression circuits 104 and 
106 with timing signals referenced to PCR. 

The decoder unit 200 is required to synchronously 
20 decode audio and video data. 

The audio and video data compression circuits 104 
and 106 (an audio-video input processing series) define 
the times of their signal inputs by using the timing 
information representative of the common reference time 
25 (PCR) (timing signals generated by the reference time 

generation circuit 103). The defined times are used as 
time management information (PTS: Presentation Time 
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Stamp). Namely, the audio and video data compression 
circuits 104 and 106 counts the sampled (or input) 
times of analog audio and video data by a 90 kHz 
counter called a PCR and supply representative values 
5 of the counts to the decoder unit 200, as PTS . 

More specifically, the audio data compression 
circuit 104 converts analog audio data applied to the 
input terminal 101 into digital audio data and 
generates PTS indicating the sampling timings of the 

10 audio data, in accordance with the timing signals 

supplied from the reference time generation circuit 
103. The audio data compression circuit 104 supplies 
the digital audio data and generated PTS to the 
compressed audio data packeting circuit 107. 

15 The audio data compression circuit 104 encodes 

(compresses) the audio data by using correlation or the 
like and generates each audio access unit (AAU) having 
a compression/expansion base unit, under the control of 
the compression control circuit 105. The audio data 

20 compression circuit 104 supplies the compressed audio 
data packeting circuit 107 with an audio elementary 
stream (AES) constituted of generated AAU's. 

In this case, the compression control circuit 105 
controls a compression factor of the audio data 

25 compression circuit 104. 

The video data compression circuit 106 converts 
analog video data applied to the input terminal 102 



into digital video data and generates PTS indicating 
the sampling timings of the video data, in accordance 
with the timing signals supplied from the reference 
time generation circuit 103. The audio data 
compression circuit 106 supplies the digital video data 
and generated PTS to the compressed video data 
packeting circuit 109- 

The video data compression circuit 106 encodes 
(compresses) the video data by using correlation or the 
like and generates each video access unit (VAU) having 
a compression/expansion base unit. The video data 
compression circuit 106 supplies the compressed video 
data packeting circuit 109 with a video elementary 
stream (VES) constituted of generated VAU 1 s . 

The compressed audio data packeting circuit 107 
packetizes AES supplied from the audio data compression 
circuit 104 under the control of the packet size 
control circuit 108 to be described later, and loads 
PTS supplied from the audio data compression circuit 
104 in the header field (PES header) of each packet 
(PES packet). A packetized elementary stream (PES) 
constituted of the PES packets is supplied to the 
multiplex transmission packeting circuit 110. 

The compressed video data packeting circuit 109 
packetizes VES supplied from the video data compression 
circuit 106 in the unit of VAU, and loads PTS supplied 
from the video data compression circuit 106 in the 
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header field of each packet. A PES constituted of the 
packets is supplied to the multiplex transmission 
packeting circuit 110. 

The multiplex transmission packeting circuit 110 
5 cuts off each PES supplied from the compressed audio 

and video data packeting circuits 107 and 109 to have a 
size suitable for the transmission path 300 and form a 
transmission packet ( TS packet). The header field ( TSP 
header) of each TS packet is loaded with program 

10 identification information ( PID (Packet Identification) 
information for identifying whether the packet is video 
or audio data), PCR supplied from the reference time 
generation circuit 103, and other information. A 
transport stream ( TS ) constituted of TS packets with 

15 the header information is supplied to the modulation 
circuit 111. 

According to ISO/IEC 13818-1, it stipulates that 
although it is not necessarily required to load PCR in 
each TS packet, PCR is required to be loaded and 

20 transmitted at least once per 100 ms. 

The modulation circuit 111 modulates TS supplied 
from the multiplex transmission packeting circuit 110 
so as to make it suitable for the characteristics of 
the transmission path 300, and supplies the modulated 

25 TS to the decoder unit 200 over the transmission path 
300. The transmission path 300 may be a recording 
medium such as a tape, a wired path, a wireless path, 
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or the like. 

Upon reception of the modulated TS, the 
demodulation circuit 201 of the decoder unit 200 
demodulates the TS and supplies the demodulated TS to 
the multiplex data separation circuit 202. 

The multiplex data separation circuit 202 checks 
PID contained in each TS packet of TS supplied from the 
demodulation circuit 201. If PID indicates audio data, 
data in a data field (pay load) of the TS packet is 
supplied to the compressed audio data depacketing 
circuit 203. If PID indicates video data, data in the 
data field of the TS packet is supplied to the 
compressed video data depacketing circuit 204. 

The multiplex data separation circuit 202 also 
derives PCR from each TS packet of TS supplied from the 
demodulation circuit 201, and supplies it to the 
reference time recovery circuit 205. 

The compressed audio data depacketing circuit 203 
extracts PTS from the data supplied from the multiplex 
data separation circuit 202, and supplies it to the 
audio data decoding circuit 206. 

The compressed audio data depacking circuit 203 
also recovers AAU from the data supplied from the 
multiplex data separation circuit 202, and supplies it 
to the audio data decoding circuit 206. 

The compressed video data depacketing circuit 204 
derives PTS from the data supplied from the multiplex 
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data separation circuit 202, and supplies it to the 
video data decoding circuit 207, 

The compressed audio data depacking circuit 204 
also recovers VAU from the data supplied from the 
5 multiplex data separation circuit 202, and supplies it 
to the video data decoding circuit 207. 

The reference time recovery circuit 205 generates 
a system clock from PGR supplied from the multiplex 
data separation circuit 202, by using a PLL or the 
q 10 like, to thereby recover PCR which was intermittently 
y] received. The reference time recovery circuit 205 

V supplies the recovered PCR to the audio and video data 

i - 

decoding circuits 206 and 207. 

rl = 

!y The audio data decoding circuit 206 decodes 

Y-£ 15 (expands) AAU supplied from the compressed audio data 

H depacketing circuit 203 to obtain digital audio data. 

if! 

The audio data decoding circuit 206 compares PTS 
supplied from the compressed audio data depacketing 
circuit 203 with PCR supplied from the reference time 

20 recovery circuit 205, and at a timing determined by the 
comparison result, converts the digital audio data into 
analog audio data which is output as an audio signal 
via the output terminal 208 . 

The video data decoding circuit 207 decodes 

25 (expands) VAU supplied from the compressed video data 

depacketing circuit 204 to obtain digital video data. 
The video data decoding circuit 207 compares PTS 
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supplied from the compressed video data depacketing 
circuit 204 with PCR supplied from the reference time 
recovery circuit 205, and at a timing determined by the 
comparison result, converts the digital video data into 
5 analog video data which is output as a video signal via 
the output terminal 209 . 

The monitor unit 400 receives the audio and video 
data decoded by the decoder unit 200 and outputs it. 

A series of operations of the digital transmission 

10 system has been described above. 

Next, packeting audio data under the control of 
the packet size control circuit, which is the most 
characteristic feature of the digital transmission 
system, will be specifically described. 

15 First, the audio data compression circuit 104 

compresses audio data at a compression factor 
designated by the compression control circuit 105 to 
thereby generate AAU, in accordance with a 
predetermined scheme (e.g., ISO/IEC 11172-3), and 

20 outputs AES constituted of generated AAU 1 s . Therefore, 
the audio data compression circuit 104 outputs AES 
constituted of, for example, AAU0, AAU1 , AAU2,... as 
shown in Fig. 3 A . At the same time, the audio data 
compression circuit 104 also outputs PTS. 

25 Next, the compressed audio data packeting circuit 

107 packetizes AES output from the audio data 
compression circuit 104, and loads PTS output from the 
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audio data compression circuit 104 in the header field 
(PES header) of each packet (PES packet). Packetizing 
at this time is executed under the control of the 
packet size control circuit 108. 
5 More specifically, the packet size control circuit 

108 determines the packet length of a PES packet 
corresponding to AES having the compression factor 
designated by the compression control circuit 105 and 
set at the audio data compression circuit 104. 
f=% 10 AAU is a data train compressed by using audio data 

fi of 24 ms as one unit. According to ISO/IEC 111723, it 

\I stipulates that although it is not necessarily required 

^ to load PTS in each PES header, PTS is required to be 

?y 

Iy loaded and transmitted at least once per 700 ms . 

Q 15 Therefore, the packet size control circuit 108 

determines "m" which makes a calculation result of the 
%Q following formula (1) have "0" or a "positive" value 

~~ nearest to "0". The packet size control circuit 108 

supplies this value "m" to the compressed audio data 
20 packeting circuit 107, as information on the number of 
AAU ! s to be packetized. 

TSd*n - (AAUd*m + Lpesh ) ... (1) 

where TSd is the number of pay loads of the TS packet, 
AAUd is the data length of AAU, Lpesh is the data 
25 length of PES, n is an arbitrary integer, and m is an 
integer which satisfies m x 24 ms < 700 ms. 

The packet size control circuit 108 supplies the 



determined "m" to the compressed audio data packeting 
circuit 107 . 

The packet size control circuit 108 may determine 
"m" by calculating the formula (1) or may determine it 
by using a table which stores a correspondence between 
each compression factor and "m" . 

Therefore, as shown in Fig. 3B, the compressed 
audio data packeting circuit 107 packetizes AAU 1 s in 
the unit of the number (in the example shown in Fig. 
3B, m = 2) of AAU's designated by the packet size 
control circuit 108, loads PTS in the header field (PES 
header) of each packet (PES packet), and outputs it as 
PES. 

The multiplex transmission packeting circuit 110 
further packetizes PES output from the compressed audio 
data packeting circuit 107, as shown in Fig. 3C, to 
have the size of a fixed length suitable for the 
transmission path 300, and outputs it as TS . 

If the calculation result of the formula (1) made 
by the packet size control circuit 108 to determine the 
number of AAU's contained in a PES packet, i.e., the 
value of "m", is not "0", the multiplex transmission 
packeting circuit 110 packetizes a PES packet into TS 
packets by adding, if necessary, staff data to the last 
TS packet in order to make the last TS packet have the 
fixed length. In this case, however, since the 
compressed audio data packeting circuit 107 packetizes 
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AES into PES packets under the control of the packet 
size control circuit 108, the amount of staff data can 
be made minimum. 

With the above configuration, a redundancy can be 
5 reduced when PES constituted of fixed length data such 
sa AAU is packet ized into a TS packet having a fixed 
length. 

The present invention may be applied to a system 
constituted of a plurality of apparatuses (e.g., a host 

10 computer, interface units, a reader, a printer, and the 
like) or to a system constituted of a single apparatus 
(e.g., a digital VTR, a digital camera, a digital 
television or the like). Namely, the invention is 
applicable to a partially or wholly integrated 

15 combination of the camera unit 10, encoder unit 100, 
decoder unit 200 and monitor unit 400. 

The scope of the invention also includes a system 
or apparatus whose computer ( CPU or MPU ) runs to 
operate various devices connected thereto in accordance 

20 with software program codes supplied to the system or 
apparatus so as to realize the functions of the above 
embodiments . 

In this case, the software program codes 
themselves realize the embodiment functions. 

25 Therefore, the program codes themselves and means for 
supplying such program codes to a computer, e.g., a 
storage medium storing such program codes, constitute 



-the present invention. 

The storage medium storing such program codes may 
be a floppy disk, a hard disk, an optical disk, a 
magneto-optical disk, a CD-ROM, a magnetic tape, a non- 
volatile memory card, a ROM, and the like. 

Obviously, such program codes are other types of 
embodiments of this invention, not only for the case 
wherein the embodiment functions are realized by 
executing the program codes supplied to the computer 
but also for the case wherein the embodiment functions 
are realized by the program codes used with an OS 
(operating system) on which the computer runs or with 
other various types of application software - 

Furthermore, the scope of the invention also 
includes obviously the case wherein in accordance with 
the program codes stored in a memory of a function 
expansion board or unit connected to the computer 
supplied with the program codes, a CPU or the like of 
the function board or unit executes part or the whole 
of the actual tasks for realizing the embodiment 
functions . 

In other words, the foregoing description of 
embodiments has been given for illustrative purposes 
only and not to be construed as imposing any limitation 
in every respect . 

The scope of the invention is, therefore, to be 
determined solely by the following claims and not 
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limited by the text of the specifications and 
alterations made within a scope equivalent to the scope 
of the claims fall within the true spirit and scope of 
the invention. 



